HDDS-15533. DNS refresh on heartbeat failure for DN to SCM#10844
Open
kerneltime wants to merge 1 commit into
Open
HDDS-15533. DNS refresh on heartbeat failure for DN to SCM#10844kerneltime wants to merge 1 commit into
kerneltime wants to merge 1 commit into
Conversation
Resolves the HDDS-15682 TODO in HostAndPort: makes the cached address threadsafe (volatile) and adds refresh() to re-resolve host:port on demand. On a connection-class heartbeat failure past a threshold (gated by ozone.client.failover.resolve-needed), HeartbeatEndpointTask asks SCMConnectionManager.refreshSCMServer to re-resolve the SCM peer and, on an IP change, rebuild the endpoint under the same host:port key. The stable key means no re-keying or StateContext queue migration is needed. New knob: hdds.heartbeat.address.refresh.threshold (default 3). Generated-by: Claude Code (Opus 4.8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds DNS refresh on the DN→SCM heartbeat path, completing the
// TODO: HDDS-15533left inHostAndPortby HDDS-15682 (#10612).#10612 keys SCM-node identity on the stable configured
host:port, but it resolves the address once and freezes it — so a datanode keeps dialing a dead IP after an SCM pod is rescheduled to a new IP. This change:HostAndPort.addressthreadsafe (volatile) and addsrefresh()to re-resolvehost:portand swap the cached address on an IP change (resolving the TODO).SCMConnectionManager.refreshSCMServer(), which rebuilds the endpoint under the samehost:portkey and closes the stale proxy. Because the key is stable, there is no re-keying, collision check, orStateContextqueue migration.HeartbeatEndpointTasktriggers the refresh on a connection-class heartbeat failure once the missed-heartbeat threshold is reached, gated by the existingozone.client.failover.resolve-neededflag.New config:
hdds.heartbeat.address.refresh.threshold(default 3). Off by default — withozone.client.failover.resolve-needed=false(the default) behaviour is unchanged.This replaces the earlier stacked attempt (szetszwo#11), rebuilt directly on
masternow that #10612 has merged; it also supersedes the closed #10488.What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-15533
How was this patch tested?
New
TestHeartbeatEndpointTaskDnsRefreshverifies a connection-class failure past the threshold triggers the refresh, and that flag-off, application-level errors, and below-threshold do not.mvn checkstyle:checkpasses onhadoop-hdds/commonandhadoop-hdds/container-service. Full module build/tests were not run locally; relying on CI.Generated-by: Claude Code (Opus 4.8)